Repository navigation
Conversation
jiaqizho
force-pushed
the
pax-open-source-lighting-4
branch
from
March 21, 2025 06:47
c47b9d8 to
10bf670
Compare
leborchuk
approved these changes
Mar 23, 2025
jiaqizho
force-pushed
the
pax-open-source-lighting-4
branch
from
April 1, 2025 05:59
10bf670 to
9d531c1
Compare
jiaqizho
force-pushed
the
pax-open-source-lighting-4
branch
5 times, most recently
from
April 1, 2025 13:14
9135cbb to
f8a730e
Compare
yjhjstz
reviewed
Apr 1, 2025
yjhjstz
reviewed
Apr 1, 2025
yjhjstz
reviewed
Apr 1, 2025
jiaqizho
force-pushed
the
pax-open-source-lighting-4
branch
5 times, most recently
from
April 2, 2025 06:11
6b9a9ca to
287abbd
Compare
tuhaihe
reviewed
Apr 2, 2025
Member
|
Hey, here are some comments for your information:
If the dir
|
jiaqizho
force-pushed
the
pax-open-source-lighting-4
branch
from
April 2, 2025 07:08
287abbd to
2164a5a
Compare
Member
|
+Comments:
|
yjhjstz
reviewed
Apr 2, 2025
yjhjstz
reviewed
Apr 2, 2025
yjhjstz
reviewed
Apr 2, 2025
yjhjstz
reviewed
Apr 2, 2025
jiaqizho
force-pushed
the
pax-open-source-lighting-4
branch
3 times, most recently
from
April 3, 2025 04:16
83f5928 to
5711495
Compare
Internal partition of PAX is no longer used, so we remove this feature from PAX. After this commit, the reloption 'partition_by' and 'partition_ranges' are also removed.
1. Due to the change of kernel behavior, there are many changes in the execution plan generated by orca. 2. Disable optimizer_trace_fallback to avoid orca fallback generating unstable output.
Several bugs are found and fixed in this commit:
1. Wrong result by filtering bpchar values with bloomfilter,
Calculating and testing ignores trailing spaces for bpchar bytes.
2. Fix index counting when copying bit values to a buffer. Increasing
the index counter no matter whether the current value is null or not.
3. Run sparse filter with group stats before reading the group.
4. Guard pax_enable_sparse_filter when initializing ParallelScanDesc.
Besides fixing the above issue, the pax tests run two pass. One turns off
vectorization, while the other one turns on vectorization.
This commit adds a new manifest implementation for catalog. The new
implementation used manifest files(regular file), see the third type
below. The interface of manifest API is saved in
contrib/pax_storage/src/cpp/catalog/manifest_api.h
We have 3 implementations for pax catalog:
1. Use the original pax catalog directly, i.e. call the catalog functions
in pax code. No intermediate interface is introduced. The catalog
table pg_ext_aux.pg_pax_tables is required.
Set USE_MANIFEST_API=OFF USE_PAX_CATALOG=ON to enable it.
2. Use the original pax catalog through manifest API. All catalog access
is done through the manifest API. The manifest API is implemented
by the original pax catalog. pg_ext_aux.pg_pax_tables is also
required.
Set USE_MANIFEST_API=ON USE_PAX_CATALOG=ON to enable it.
3. Use manifest files to manage catalog for PAX through manifest API.
All catalog access is done through the manifest API. The original
catalog pg_ext_aux.pg_pax_tables is no more required. The per-table
auxiliary table is also changed from storing micro partition info
to saving the path of manifest file path.
Set USE_MANIFEST_API=ON USE_PAX_CATALOG=OFF to enable it.
Each pax table now uses a single manifest file to store the catalog
indicating all micro partition info. The design disallows to write,
i.e. insert/delete/update, concurrently. To avoid concurrent write,
a heavy lock must be taken before writing. The steps of accessing
catalog is:
1. Build the auxiliary table name by the oid of pax table.
2. Open the auxiliary table of the pax table, and fetch the path
for manifest file. The auxiliary table has only one effective
tuple.
3. Open the manifest file and deserialize content to json object.
4. Access manifest API and return result by the internal manifest
json object.
This commit adds Apache license 2.0 to all source files and header files in PAX.
In PAX `PORC` format, length streaming is used to record the length of each DATUM in a non-fixed-length column. The composition of length streaming is equivalent to the length array, and its size equal to the number of rows. When reading thenon-fixed-length column, PAX needs to use the length array to calculate the offset array in advance, the offset array can help the format reader quickly locate the middle row. In commit "Performance/improve pax insert performance", PAX no longer builds offsets array during the write phase, which actually breaks the assumption of column: only some column methods distinguish between read and write paths. In the current commit, the length streaming of PAX is changed to the offset streaming. - On the read path, non-fixed-length column no need to build the offset array - On the read path, only using the offset array is more cache-friendly - On the write path, only the offset array needs to be built. And the performance is comparable to building the length array Offset streaming also has disadvantages: the compression rate of offset streaming is likely to be lower than that of length streaming. Currently, PAX does not support DELTA encoding. Once DELTA encoding is supported, this disadvantages may can resolved.
PAX is no longer support object storage in lightning. The implementation of RemoteFileSystem is moved from lightning to cloud to support object storage access by the abstract file API defined in PAX. Despite pax doesn't support RemoteFileSystem, we still disallow to use the dfs tablespace for PAX table.
1. the toast table of the auxiliary table should also be in the pg_aux_ext namespace 2. use GetCatalogSnapshot() as a snapshot when querying auxiliary tables
A doc directory has been added to the PAX project, which will contain documentation for the modules in PAX. - Introduce - Project description - Meta data - Storage format - Toast - Clustering - Filter
After commit(588f5c9) and commit(ca9379e). Access method adds two callbacks that must be implemented. - relation_get_block_sequences: Returns the block sequences contained in this relation. See BlockSequence for details. Currently used by BRIN. - relation_get_block_sequence: Determines the block sequence in which the logical heap 'blkNumber' falls. See BlockSequence for details. Currently used by BRIN. Currently, PAX does not support brin index. So these AM method have been added in the current commit but have not been implemented. After CBDB cherry-picks the complete changes of brin index, consider making PAX support Brin index.
In prev commit(8cf1aba) removed the `am->swap_relation_files` in function `swap_relation_files`. This will cause problems in rewrite table case for the custom AM(like PAX). ex. ``` CREATE TABLE list_parted (a numeric, b int, c int8) PARTITION BY list (a) using pax; CREATE TABLE sub_parted PARTITION OF list_parted for VALUES in (1) PARTITION BY list (b); CREATE TABLE sub_part1(b int, c int8, a numeric) DISTRIBUTED BY (a); ALTER TABLE sub_parted ATTACH PARTITION sub_part1 for VALUES in (1); CREATE TABLE sub_part2(b int, c int8, a numeric) distributed by (a); ALTER TABLE sub_parted ATTACH PARTITION sub_part2 for VALUES in (2); INSERT into list_parted VALUES (2,5,50); INSERT into list_parted VALUES (3,6,60); INSERT into sub_parted VALUES (1,1,60); INSERT into sub_parted VALUES (1,2,10); ALTER TABLE list_parted SET DISTRIBUTED BY (c); select * from list_parted; -- wrong result ``` The `Alter DISTRIBUTED BY` will rewrite the data into a temp table and exchange the relfilenode in temp table with origin table. But without `am->swap_relation_files` call, some of meta or data won't be swap.
jiaqizho
force-pushed
the
pax-open-source-lighting-4
branch
from
April 10, 2025 06:34
3eabb10 to
3a0b78d
Compare
After CBDB revert the '64 bit relfilenode', PAX still need adapt the change.
In PAX, The naming convention for visibility maps is: <blocknum>_<generation>_<tag>.visimap - `blocknum` is the current data file name - `generation` is the current visimap generation number. Each deletion of this data file will increase the generation number by 1 - `tag` is the current transaction id. This field is used to ensure the uniqueness of the visimap file name. When the USE_ASSERT_CHECKING is undefined, `generation` cannot be incremented. So if we in the same transaction and update same row twice, then PAX will open the same `.visimap` file.
PAX was based on an older Cloudberry version with copied/modified regression tests in `contrib/pax_storage/src/test/regress/` Updated PAX regression tests to align with the latest version and fixed failed cases: - Synchronized PAX regression tests with current test suite(src/test/regress/) - Fixed ORCA plan differences caused by cherry-picking features: Dynamic Index/Bitmap/Seq Scan, multi-groupset, query parameters, and so on - Resolved planner plan diff - Addressed result diff by marking unsupported test cases
PAX was based on an older Cloudberry version with copied/modified isolation2 tests in `contrib/pax_storage/src/test/isolation2/` Updated PAX isolation2 tests to align with the latest version and fixed failed cases: - Synchronized PAX isolation2 tests with current test suite(src/test/isolation2/) - Change the `uao` test case to PAX which is used to running with AO/AOCS - Remove the unused test cases(like check the gp_aoseg, gp_fastsequence ...) - Fix the plan diffs
The extension vectorization is not open source yet, and the open source version of PAX has removed vectorization-related test cases.
Previously PAX used the internal gitlab repository as a submodule. Now it has switched to using the github repository.
Manifest metadata implementation in PAX is still an experimental feature. Current commit changes the default catalog implementation to the auxiliary table implementation. Also, PAX tests are added to cloudberry's CI
jiaqizho
force-pushed
the
pax-open-source-lighting-4
branch
from
April 10, 2025 06:59
3a0b78d to
e52bd47
Compare
Removed some useless file and change fix the pax icw_test in github CI Also changed the `Cloudberry Database` to `Apache Cloudberry`
jiaqizho
force-pushed
the
pax-open-source-lighting-4
branch
from
April 10, 2025 07:02
e52bd47 to
cdc18d3
Compare
This was referenced Apr 10, 2025
This was referenced Apr 15, 2025
1 of 13 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #ISSUE_Number
What does this PR do?
Type of Change
Breaking Changes
Test Plan
make installcheckmake -C src/test installcheck-cbdb-parallelImpact
Performance:
User-facing changes:
Dependencies:
Checklist
Additional Context
CI Skip Instructions